APP19 Recovery (Firm Switch DAS Fix Engine)

APP19 Purpose:

APP19 Fix Engine receives MESSAGING messages from MESSAGING sending SITE 1 and sends them to APP01 via OMS for actual trading.

APP19 Fix Engines can also send to APP02, APP03 or APP04 if SITE 1 route them there using fix tags, but typically they are sent to APP01.

APP19 Fix Engines then receive related responses from these services for processing and/or error handling.

Use the following hyperlinks to jump to the desired section of APP19 documentation:

APP19_Recovery_Considerations

APP19_NTM_Control_Commands

APP19_Troubleshooting_Table

APP19_Monitoring_Considerations

APP19 Recovery Considerations:

Stopping/Restart Processes:

- Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

- When moving between nodes:

1) ALTERNATE NODES are not defined for APP19 services.

2) DR NODES must be allocated un-natted nodes.

No node can support more than one natted address at the same time.

- When stopping/restarting APP19 processes:

1) Notify MESSAGING sending SITE 1 involved and work in cooperation with them, as appropriate to situations.

2) Stop the APP19 process.

3) If NOT moving APP19 to new node, skip to step 5.

a) If moving the APP19 to a new node, copy the day’s APP19 FIX engine files to alternate node:

a) Copy: \\data\{APP19}\*.log, *.body, *.header, *.seqnums, *.session file created for the day to the alternate node.

b) Copy: \\data\{APP19}\Global.* file created for the day to the alternate node.

c) If the target folder does not exist on the new node, you must first either create the folder, or copy the entire folder.

d) If these files are not moved before the process restarts on the new node, there will be a chance of sequence number miscommunications between the MESSAGING sending firm involved and .

4) Start the APP19 process.

5) Open the channel for the process affected and confirm MESSAGING sending firm connects as expected.

APP19 NTM Control Commands:

Open OSF Channel:

- Use NTM Control Utility – Service Control - APP19 – Open OSF Channel to make OSF connection possible.

Close OSF Channel:

- Use NTM Control Utility – Service Control - APP19 – Close OSF Channel to make OSF connection impossible.

Set Inbound Sequence Number:

- Use NTM Control Utility – Service Control - APP19 – Set Inbound Sequence Number to set Inbound Sequence Number.

Set Outbound Sequence Number:

- Use NTM Control Utility – Service Control - APP19 – Set Outbound Sequence Number to set Outbound Sequence Number.

Enable 29West Stats:

- Use NTM Control Utility – Service Control - APP19 – Enable 29West Stats to start collection and display of LBM related stats.

Disable 29West Stats:

- Use NTM Control Utility – Service Control - APP19 – Disable 29West Stats to stop collection and display of LBM related stats.

APP19 Troubleshooting Table:

APP19 Symptom	Impacts	Response
Firm disconnects or Logs out of session Evidenced by: - EMT message saying {firm} is disconnected and/or {firm} is logged out. - Stats monitor shows disconnected in status column. {Firm} will be further identified in EMT message by including “LocalFixId” and “RemoteFixID” as configured in APP19Services.xml file within the disconnect message.	APP19 is no longer able to receive MESSAGING messages or return responses to MESSAGING sending SITE 1.	1) Contact firm. 2) Work with firm and/or Technical Services as necessary to isolate cause of issues and resolve them. 3) Stop/Restarts of affected application service may help resolve the issue.

APP19 Monitoring Considerations:

Stats Monitors: APP19 App Connect Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate communications from MESSAGING Sending SITE 1 to fix engine for special routing (via OMS to APP01, Away Market Destinations or APP04). Monitor shows connection status between fix engine and firm, or of fix engine and application.	PROD MENU: SITE 1witch Monitoring Menu To Exit: Close Window	- Color of data in columns - Status, - Write Queue	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Status is Disconnected or Open	Firm is not connected. 1) Use NTM Control Utility APP19 Service Controls to Open Channels. 2) Call Production Control if needed.
			Write Queue is non-zero values and not reducing as expected.	Firm may not be processing as expected. 1) Call firm and work with Technical Services if necessary.

Stats Monitors: APP19 App Processing Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate communications from MESSAGING Sending SITE 1 to fix engine for special routing (via OMS to APP01, Away Market Destinations or APP04). Monitor shows processing statistics between fix engine and MESSAGING Sending Firm.	PROD MENU: SITE 1witch Monitoring Menu To Exit: Close Window	- Color of data in columns - InCount, - OutCount, - LastInTime, - LastOutTime	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			No data is displayed.	No data has been generated between MESSAGING Sending Firm and fix engine. 1) Check APP19 Fix message files. If all sizes are zero, no traffic has been generated. 2) Test sending MESSAGING to test FIX engine.
			Aggregate InCount (by service name) is not less than, or equal to aggregate (by service name) OutCount. NOTE: LastOutTime (by service name) should also be later than LastInTime (by service name)	Firm may not be processing as expected. 1) Check APP19 FIX message files to confirm inbound messages match outbound messages. 2) Call firm and work with Technical Services if necessary.

Stats Monitors: APP19 IPC Instance Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate communications from MESSAGING Sending SITE 1 to fix engine for special routing (via OMS to APP01, Away Market Destinations or APP04). Monitor shows IPC channel processing statistics.	PROD MENU: SITE 1witch Monitoring Menu To Exit: Close Window	- Color of data in columns - Service, - Hostname, - Msgs In, - Msgs Out	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APP02 processes are displayed as expected.	APP02 Service has not been started. 1) Check status of service.
			Msgs In and/or Msgs Out are zero.	No messages have been sent/received since that monitor has been started. 1) Check APP19 log files. 2) Call firm and work with Technical Services if necessary.

Stats Monitors: APP19 to OMS IPC Connect Queues	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate communications from MESSAGING Sending SITE 1 to fix engine for special routing (via OMS to APP01, Away Market Destinations or APP04). Monitor shows IPC channel connectivity status to OMS processes.	PROD MENU: SITE 1witch Monitoring Menu To Exit: Close Window	- Color of data in columns - Source, - Dest, - Status, - Queue Size, - Msgs Out/Sec	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APP19 or OMS processes are displayed as expected.	APP19 or OMS Service has not been started or hasn't processed any messages since monitor has been started. 1) Check status of service. 2) Check APP19 or OMS log files.
			Status is not CONNECTED.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process. 2) Notify Production Support if issues.
			Queue size is non-zero value and not decreasing as expected.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process. 2) Notify Production Support if issues.

Stats Monitors: APP19 to FMR IPC Connect Queues	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate communications from MESSAGING Sending SITE 1 to fix engine for special routing (via OMS to APP01, Away Market Destinations or APP04). Monitor shows IPC channel connectivity status to FMR processes.	PROD MENU: SITE 1witch Monitoring Menu To Exit: Close Window	- Color of data in columns - Source, - Dest, - Status, - Queue Size, - Msgs Out/Sec	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APP19 or FMR processes are displayed as expected.	APP19 or FMR Service has not been started or hasn't processed any messages since monitor has been started. 1) Check status of service. 2) Check APP19 or FMR log files.
			Status is not CONNECTED.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process. 2) Notify Production Support if issues.
			Queue size is non-zero value and not decreasing as expected.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process. 2) Notify Production Support if issues.